Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination.

PURPOSE: The Korea Medical Licensing Exam (KMLE) typically contains a large number of items. The purpose of this study was to investigate whether there is a difference in the cut score between evaluating all items of the exam and evaluating only some items when conducting standard-setting. METHODS: We divided the item sets that appeared on 3 recent KMLEs for the past 3 years into 4 subsets of each year of 25% each based on their item content categories, discrimination index, and difficulty index. The entire panel of 15 members assessed all the items (360 items, 100%) of the year 2017. In split-half set 1, each item set contained 184 (51%) items of year 2018 and each set from split-half set 2 contained 182 (51%) items of the year 2019 using the same method. We used the modified Angoff, modified Ebel, and Hofstee methods in the standard-setting process. RESULTS: Less than a 1% cut score difference was observed when the same method was used to stratify item subsets containing 25%, 51%, or 100% of the entire set. When rating fewer items, higher rater reliability was observed. CONCLUSION: When the entire item set was divided into equivalent subsets, assessing the exam using a portion of the item set (90 out of 360 items) yielded similar cut scores to those derived using the entire item set. There was a higher correlation between panelists' individual assessments and the overall assessments.

Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination.

PURPOSE: This study explored the possibility of using the Angoff method, in which panel experts determine the cut score of an exam, for the Korean Nursing Licensing Examination (KNLE). Two mock exams for the KNLE were analyzed. The Angoff standard setting procedure was conducted and the results were analyzed. We also aimed to examine the procedural validity of applying the Angoff method in this context. METHODS: For both mock exams, we set a pass-fail cut score using the Angoff method. The standard setting panel consisted of 16 nursing professors. After the Angoff procedure, the procedural validity of establishing the standard was evaluated by investigating the responses of the standard setters. RESULTS: The descriptions of the minimally competent person for the KNLE were presented at the levels of general and subject performance. The cut scores of first and second mock exams were 74.4 and 76.8, respectively. These were higher than the traditional cut score (60% of the total score of the KNLE). The panel survey showed very positive responses, with scores higher than 4 out of 5 points on a Likert scale. CONCLUSION: The scores calculated for both mock tests were similar, and were much higher than the existing cut scores. In the second simulation, the standard deviation of the Angoff rating was lower than in the first simulation. According to the survey results, procedural validity was acceptable, as shown by a high level of confidence. The results show that determining cut scores by an expert panel is an applicable method.

Comparison of standard-setting methods for the Korea Radiological technologist Licensing Examination : Angoff, Ebel, Bookmark, and Hofstee.

PURPOSE: This study aims to compare the various standard setting methods for the Korean Radiological Technologist Licensing Examination with the fixed cut score and suggest the most appropriate method. METHODS: Six radiological technology professors, set the standards of 250 items for Korean Radiological Technologist Licensing examination that were conducted on December 2016 by using Angoff, Ebel, bookmark, and Hofstee methods. RESULTS: With the maximum percentile score of 100, the cut score for the examination was 71.27 in Angoff method, 62.2 in Ebel method, 64.49 in bookmark method, and 62 in Hofstee. Based on the Hofstee's acceptable cut score, the acceptable cut score for the examination was between 52.83 and 70, but the cut score was 71.27 in Angoff method. CONCLUSION: Above results suggested that the best standard setting methods to determine the cut score was panel discussion with the modified Angoff or Ebel methods, and verification of the rated results by Hofstee method. Because there was still no adoption of standard setting in the Korean Radiological Technologist Licensing Examination, this study will be able to provide the practical guideline to introduce the standard setting.

The relationship of individual examinee characteristics and acceptability of smart device-based testing to test score in the practice test of the Korea Emergency Medicine Technician Licensing Examination.

PURPOSE: Smart device-based testing (SBT) began to be introduced in the Republic of Korea's high-stakes examination system, starting with the Korea Emergency Medicine Technician Licensing Examination (KEMTLE) in December 2017. In order to minimize how variation in examinees' environment may affect the test score, it aimed to identify any variables related to individual characteristics and acceptability that are related to the examinees' test scores in the SBT practice test. METHODS: Out of 569 candidate students who were administered the KEMTLE on September 12, 2015, 560 students responded to the survey questionnaire on the acceptability of SBT after the examination. The questionnaire addressed 8 individual characteristics: 2 satisfaction, 9 convenience, and 9 preference items. Comparison analysis between individual variables was performed. Furthermore, generalized linear model (GLM) analysis to find the effect of individual characteristics and acceptability of SBT on test score were conducted. RESULTS: Among those who preferred SBT compared to paper-and-pencil testing, test scores were higher for male participants (M=4.36, SD=0.72) than for female participants (M=4.21, SD=0.73). According to GLM, no variables evaluated, including gender, experiences of CBT(Computer based test), SBT, and using a tablet PC, were statistically significantly different by total score, scores on multimedia items, or scores on text items. CONCLUSION: The variables of individual characteristics and acceptability of SBT did not affect the SBT practice test scores of emergency medicine technician students in Korea. Adoption of SBT for the KEMTLE should be possible to execute without interference from the variables examined in this study.

Perception survey on the introduction of clinical performance examination as part of the national nursing licensing examination in Korea.

PURPOSE: The purpose of this study was to analyze opinions about the action plan for implementation of clinical performance exam as part of the national nursing licensing examination and presents the expected effects of the performance exam and aspects to consider regarding its implementation. METHODS: This study used a mixed-methods design. Quantitative data were collected by a questionnaire survey, while qualitative data were collected by focus group interviews with experts. The survey targeted 200 nursing professors and clinical nurses with more than 5 years of work experience, and the focus group interviews were conducted with 28 of professors, clinical instructors, and nurses at hospitals. RESULTS: First, nursing professors and clinical specialists agreed that the current written tests have limitations in evaluating examinees' ability, and that the introduction of a clinical performance exam will yield positive results. Clinical performance exam is necessary to evaluate and improve nurses' work ability, which means that the implementation of a performance exam is advisable if its credibility and validity can be verified. Second, most respondents chose direct performance exams using simulators or standardized patients as the most suitable format of the test. CONCLUSION: In conclusion, the current national nursing licensing exam is somewhat limited in its ability to identify competent nurses. Thus, the time has come for us to seriously consider the introduction of a performance exam. The prerequisites for successfully implementing clinical performance exam as part of the national nursing licensing exam are a professional training process and forming a consortium to standardize practical training.

Smart device-based testing for medical students in Korea: satisfaction, convenience, and advantages.

The aim of this study was to investigate respondents' satisfaction with smart device-based testing (SBT), as well as its convenience and advantages, in order to improve its implementation. The survey was conducted among 108 junior medical students at Kyungpook National University School of Medicine, Korea, who took a practice licensing examination using SBT in September 2015. The survey contained 28 items scored using a 5-point Likert scale. The items were divided into the following three categories: satisfaction with SBT administration, convenience of SBT features, and advantages of SBT compared to paper-and-pencil testing or computer-based testing. The reliability of the survey was 0.95. Of the three categories, the convenience of the SBT features received the highest mean (M) score (M= 3.75, standard deviation [SD]= 0.69), while the category of satisfaction with SBT received the lowest (M= 3.13, SD= 1.07). No statistically significant differences across these categories with respect to sex, age, or experience were observed. These results indicate that SBT was practical and effective to take and to administer.

Reforms of the Korean Medical Licensing Examination regarding item development and performance evaluation.

PURPOSE: The Korean Medical Licensing Examination (KMLE) has undergone a variety of innovative reforms implemented by the National Health Personnel Licensing Examination Board (NHPLEB) in order to make it a competency-based test. The purpose of this article is to describe the ways in which the KMLE has been reformed and the effect of those innovations on medical education in Korea. METHODS: Changes in the KMLE were traced from 1994 to 2014 by reviewing the adoption of new policies by the NHPLEB and the relevant literature. RESULTS: The most important reforms that turned the examination into a competency-based test were the following: First, the subjects tested on the exam were revised; second, R-type items were introduced; third, the proportion of items involving problem-solving skills was increased; and fourth, a clinical skills test was introduced in addition to the written test. The literature shows that the above reforms have resulted in more rigorous licensure standards and have improved the educational environment of medical schools in Korea. CONCLUSION: The reforms of the KMLE have led to improvements in how the competency of examinees is evaluated, as well as improvements in the educational system in medical schools in Korea.

Exploration of examinees' traits that affect the score of Korean Medical Licensing Examination.

PURPOSE: It aims to identify the effect of five variables to score of the Korean Medical Licensing Examinations (KMLE) for three consecutive years from 2011 to 2013. METHODS: The number of examinees for each examination was 3,364 in 2011 3,177 in 2012, and 3,287 in 2013. Five characteristics of examinees were set as variables: gender, age, graduation status, written test result (pass or fail), and city of medical school. A regression model was established, with the score of a written test as a dependent variable and with examinees' traits as variables. RESULTS: The regression coefficients in all variables, except the city of medical school, were statistically significant. The variable's effect in three examinations appeared in the following order: result of written test, graduation status, age, gender, and city of medical school. CONCLUSION: written test scores of the KMLE revealed that female students, younger examinees, and first-time examinees had higher performances.

The school effect on the reliability of clinical performance examination in medical schools.

PURPOSE: The purpose of this study is to test the reliability of the clinical performance examination (CPX) using Generalizability theory (G-theory). Through G-theory, the effects of not only students and tasks but also the school will be analyzed as primary sources of error, which can affect the interpretation of the reliability of the CPX. METHODS: One thousand three hundred nineteen students from 16 medical schools that participated in the Seoul-Gyeonggi CPX Consortium 2008 were enrolled. In our research design, we suppose that student is nested within school and crossed with task. Data analysis was conducted with urGenova. RESULTS: According to our analysis, the percentage of error variance was 6.2% for school, 14.9% for student nested within school, 14.4% for task, and 3% for interaction between school and task. An effect of school on students was observed, but the interaction between task and school was insignificant. When student is nested within school, the universe score decreased and the g-coefficient was less than the g-coefficient of the p x t (p: studentm, t: task) design. CONCLUSION: The results show that generalizability theory is useful in detecting various error components in the CPX. Using the generalizability theory to improve the technical quality of performance assessments provides us with greater information compared with traditional test theories.

Test equating of the Medical Licensing Examination in 2003 and 2004 based on the item response theory.

The passing rate of the Medical Licensing Examination has been variable, which probably originated from the difference in the difficulty of items and/or difference in the ability level of examinees. We tried to explain the origin of the difference using the test equating method based on the item response theory. The number of items and examinees were 500, 3,647 in 2003 and 550, 3,879 in 2004. Common item nonequivalent group design was used for 30 common items. Item and ability parameters were calculated by three parametric logistic models using ICL. Scale transformation and true score equating were executed using ST and PIE. The mean of difficulty index of the year 2003 was -0.957 (SD 2.628) and that of 2004 after equating was -1.456 (SD 3.399). The mean of discrimination index of year 2003 was 0.487 (SD 0.242) and that of 2004 was 0.363 (SD 0.193). The mean of ability parameter of year 2003 was 0.00617 (SD 0.96605) and that of year 2004 was 0.94636 (SD 1.32960). The difference of the equated true score at the same ability level was high at the range of score of 200-350. The reason for the difference in passing rates over two consecutive years was due to the fact that the Examination in 2004 was easier and the abilities of the examinees in 2004 were higher. In addition, the passing rates of examinees with score of 270-294 in 2003, and those with 322-343 in 2004, were affected by the examination year.